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1 CAN Learn® Pre-Algebra and Algebra 


Program Description^ The / can Learn® Education System is an interactive, self-paced, 

mastery-based software system that Includes the / CAN Learn® 
Fundamentals of Math (5th-6th grade math) curriculum, the 
/ CAN Learn® Pre-Algebra curriculum, and the / CAN Learn® 
Algebra curriculum. College algebra credit is also available to 


students in participating schools through the 121 lesson CLEP 
program, an open enrollment dual-credit program for middle 
and high school students. Studies included In this WWC review 
assess the effectiveness of the Pre-Algebra and Algebra compo- 
nents of the / CAN Learn® Education System. 


Research One study of / CAN Learn® Pre-Algebra and Algebra meets 

What Works Clearinghouse (WWC) evidence standards and four 
studies meet WWC evidence standards with reservations. The 
five studies included 16,519 eighth-grade students from middle 
schools in California, Florida, Georgia, and Louisiana.^ 


Based on these five studies, the WWC considers the extent 
of evidence for / CAN Learn® Pre-Algebra and Algebra to be 
medium to large for math achievement. 



Eff6CtiV6n6SS / CAN Learn® Pre-Algebra and Algebra was found to have positive effects on math achievement. 



Math achievement 

Rating of effectiveness Positive effects 

Improvement index'^ Average: +5 percentile points 

Range: -7 to +16 percentile points 

1. This report has been updated to include reviews of 15 studies that have been reieased since 2005. Of the additionai studies, eight were not within the 
scope of the protocoi, and seven were within the scope of the protocol but did not meet evidence standards. A compiete iist and disposition of aii stud- 
ies reviewed is provided in the references. Additionaiiy, one study that met standards with reservations in the previous version (Kirby, 2005c) wili now be 
eiigibie for review as part of the WWC high school math topic area. (The protocoi for the middie school math topic area was revised to narrow the scope 
from examining any students in grades 6 to 9 to examining oniy those students who are attending middie schoois or junior high schoois. Studies examin- 
ing students in grade 9 who are attending high schooi are inciuded in the high schooi math topic area). 

2. The descriptive information for this program was obtained from a pubiiciy-avaiiable source: the program’s website (http://www.icaniearn.com/defauit.asp, 
downioaded August 2008). The WWC requests deveiopers to review the program description sections for accuracy from their perspective. Further verifica- 
tion of the accuracy of the descriptive information for this program is beyond the scope of this review. 

3. The evidence presented in this report is based on avaiiable research. Findings and conciusions may change as new research becomes avaiiabie. 

4. These numbers show the average and range of student-ievei improvement indices for ali findings across the studies. 
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Additional program Developer and contact 

information JRL Enterprises, Inc. developed and distributes / CAN Learn®. 

Address: 400 Poydras Street, Suite 1000, New Orleans, LA 70130. 
Email: info@icanlearn.com. Web: http://www.icanlearn.com. 
Telephone: (504) 263-1380. 

Scope of use 

Between 1995 (when it was first implemented) and 2008, the 
/ CAN Learn® system has been used by almost one million stu- 
dents in elementary, middle, junior high, and senior high schools, 
and community colleges across the United States. 

Teaching 

The / CAN Learn® Fundamentals of Math curriculum contains 
121 lessons with more than 40 hours of instructional video, and 
the Pre-Algebra, Algebra and Geometry curricula contain 424 
lessons with more than 120 hours of instructional video. Custom 
curriculum alignment to state, district, and school standards is 
accomplished by selecting appropriate lessons from the / CAN 
Learn® Lesson Database, which contains over 500 multimedia 
lessons. The curricula can be used online via the Internet or 
through school LANs or WANs, provided that the courseware 
is installed on a local server. 



Teachers can manage their classrooms through the / CAN 
Learn® Classroom Explorer Class Management System, which 
keeps track of student attendance, homework, and test grades. 

It can also help in developing individual learning plans to meet 
diverse student needs. A one-to-one student-to-computer ratio 
and one-on-one interaction with the classroom teacher lets each 
student progress at his or her own pace. 

Cost 

The cost of an / CAN Learn® system depends on its configura- 
tion and terms of support. Using a school’s existing hardware, 
individual subscriptions allowing access to more than 500 les- 
sons cost $43.48 per student. Varying support plans including 
training, professional development, curriculum alignments, 
implementation planning, and other pedagogical support are 
available and encouraged with a cost range from $400 to 
$20,000 per year. A complete traditional classroom installation 
of hardware and software is available, which includes 30 work- 
stations with all curriculum and class management software, 
computer hardware, network wiring, furniture, and three years 
of comprehensive onsite educational support. The cost for this 
traditional classroom installation is $200,000; the cost for a 
laptop cart configuration is $170,000. 



Research Twenty-seven studies reviewed by the WWC investigated the 
effects of / CAN Learn® Pre-Algebra and Algebra. One study 
(Kirby, 2006) was a randomized controlled trial that meets WWC 
evidence standards. Four studies (Kerstyn, 2001; Kerstyn, 2002; 
Kirby, 2004a; Kirby, 2004b) are randomized controlled trials or 
quasi-experimental designs that meet WWC evidence standards 
with reservations. The remaining 22 studies do not meet either 
WWC evidence standards or eligibility screens. 

Meets evidence standards 

Kirby (2006) assessed the impact of the / CAN Learn® system 
on math achievement in Crieans Parish schools in New Crieans, 
LA that randomly assigned students to / CAN Learn® or control 



classes during the 2003-04 school year. The study included 
2,400 eighth-grade regular education students from 13 schools 
(1,082 / CAN Learn® students and 1,318 traditional mathematics 
students). / CAN Learn® classes were compared with classes 
using a traditional curriculum. 

Meet evidence standards with reservations 

Kerstyn (2001) conducted a classroom matched-pairs quasi- 
experimental design to investigate the effect of the first year of 
implementation of the / CAN Learn® system on math achieve- 
ment of eighth-grade students in Hillsborough County Public 
Schools in Florida. At the beginning of the 2000-01 school year, 
58 / CAN Learn® classes (1,222 students) were matched with 
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58 traditional mathematics classes (1,314 students). The / CAN 
Learn® system was implemented with four separate samples of 
students enrolled in four math courses: Algebra 1 (8 classes, 

175 students). Algebra 1 Honors (8 classes, 150 students), MJ-3 
pre-algebra (32 classes, 678 students), and MJ-3 Advanced (10 
classes, 219 students). The effectiveness of the / CAN Learn® 
system is reported separately for each of the four study samples 
in the Findings section. 

Kerstyn (2002) continued the investigation of the / CAN Learn® 
system during the second year of implementation in Hillsborough 
County Public Schools with a quasi-experimental study of a 
different sample of 1,871 eighth-grade students in 129 / CAN 
Learn® classes compared to 9,254 eighth-grade students in 468 
traditional classes. The / CAN Learn® system was implemented 
with four separate samples of students enrolled in four math 
courses; Algebra 1 (18 classes, 231 students), Algebra 1 Honors 
(10 classes, 188 students), MJ-3 pre-algebra (64 classes, 1,028 
students), and MJ-3 Advanced (37 classes, 424 students). 

The students in these / CAN Learn® classes were compared 
to students enrolled in four corresponding math courses that 
were taught using a traditional instruction method: Algebra 1 
(48 classes, 964 students), Algebra 1 Honors (81 classes, 1,706 
students), MJ-3 pre-algebra (264 classes, 4,929 students), and 
MJ-3 Advanced (75 classes, 1,655 students). The effectiveness 
of the / CAN Learn® system is reported separately for each of the 
four study samples in the Findings section. 

Kirby (2004a) assessed the impact of the / CAN Learn® 
system on math achievement in a California middle school that 
randomly assigned students either to / CAN Learn® classes or 
comparison classes during the 2003-04 school year. The study 
included 204 eighth-grade students (91 students taught by one 
intervention teacher and 113 students taught by two comparison 
teachers) in Bret Harte Middle School in Alameda County, CA.® 



The intervention teacher used the pre-algebra / CAN Learn® 
mathematics curriculum.The comparison teachers used the 
state-adopted Glencoe pre-algebra textbook. Because there 
was only one / CAN Learn® teacher, it is not possible to separate 
the effect of the teacher from the effect of the / CAN Learn® 
system. This study meets WWC standards with reservations 
because of this confound. 

Kirby (2004b) assessed the impact of / CAN Learn® on math 
achievement in a northwestern Georgia middle school that 
randomly assigned students either to / CAN Learn® classes 
or comparison classes during the 2003-04 school year. The 
study included 254 eighth-grade students (91 students in / CAN 
Learn® classes and 163 students in traditional classes) in the 
Gilmer County School District. In the / CAN Learn® classes, one 
teacher facilitated instruction using the computerized curriculum. 
Students in the comparison classroom used a traditional math 
curriculum delivered by their teachers. Because there was only 
one / CAN Learn® teacher, it is not possible to separate the 
effect of the teacher from the effect of the / CAN Learn® system. 
This study meets WWC standards with reservations because of 
this confound. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or medium to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme). The extent of 
evidence takes into account the number of studies and the 
total sample size across the studies that meet WWC evidence 
standards with or without reservations.® 

The WWC considers the extent of evidence for / CAN Learn® 
Pre-Algebra and Algebra to be medium to large for math 
achievement. 



5. The study authors provided the WWC with the number of teachers in each condition. 

6. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the Intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for / CAN Learn'^ Pre-Algebra and Algebra is in Appendix A6. 
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Effectiveness 



Findings 

The WWC review of interventions for Middle School Math 
addresses student outcomes in the math achievement 
domain. The findings below present the authors’ estimates 
and WWC-calculated estimates of the size and the statistical 
significance of the effects of / CAN Learn® Pre-Algebra and 
Algebra on students.^ 

Math Achievement 

Kirby (2006) reported a positive and statistically significant effect 
of / CAN Learn® on the math exam from the Louisiana Educa- 
tional Assessment Program (LEAP) test. The WWC confirmed 
this finding after correcting the statistical significance level for 
clustering. The effect size was large enough to be considered 
substantively important according to WWC criteria (an effect size 
at least 0.25). 

Kerstyn (2001) reported positive but not statistically significant 
effects of / CAN Learn® on the Florida Comprehensive Assess- 
ment Test (FCAT) math exam for Algebra 1, Algebra 1 Honors, 
MJ-3 pre-algebra, and MJ-3 Advanced courses. The effect 
size for each of these samples was not large enough to be 
considered substantively important according to WWC criteria 
(an effect size at least 0.25). 

Kerstyn (2002) reported a positive and statistically significant 
effect of / CAN Learn® on the FCAT math exam for students in 
MJ-3 pre-algebra classes. The WWC confirmed this finding. 

The author also reported negative but not statistically significant 
effects for students in the Algebra 1, Algebra 1 Honors, and 
MJ-3 Advanced courses. The WWC confirmed that these 



negative effects were neither statistically significant nor large 
enough to be considered substantively important by WWC crite- 
ria (an effect size of at least 0.25). Thus, / CAN Learn® showed 
a statistically significant positive effect for MJ-3 pre-algebra stu- 
dents and indeterminate effects for Algebra 1, Algebra 1 Honors, 
and MJ-3 Advanced students.® 

Kirby (2004a) reported a positive and statistically significant 
effect of / CAN Learn® on the General Mathematics exam from 
the California Standards Test. The statistical significance of this 
effect was confirmed by WWC analysis. The effect size was large 
enough to be considered substantively important according to 
WWC criteria (an effect size of at least 0.25). 

Kirby (2004b) reported a positive and statistically significant 
effect of / CAN Learn® on the Math exam from the Georgia 
Criterion-Referenced Competency Test. The statistical sig- 
nificance of this effect was confirmed by WWC analysis. The 
effect size was large enough to be considered substantively 
important according to WWC criteria (an effect size of at 
least 0.25). 

In sum, in the math achievement domain, the WWC reviewed 
findings from 11 samples reported in five studies.® Four of 
these samples showed statistically significant positive effects, 
and the remaining seven samples showed indeterminate 
effects. One of the samples was examined in a study that used 
a strong design. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 



7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to 
calculate the statistical significance, see Technical Details of WWC-Conducted Computations. In the cases of Kerstyn (2002) and Kirby (2004a; 2004b), 
no corrections for clustering or multiple comparisons were needed. In the cases of Kerstyn (2001) and Kirby (2006), a correction for clustering was 
needed, so the significance levels may differ from those reported in the original study. 

8. Findings for subgroups, such as MJ-3 pre-algebra students who were not Florida Comprehensive Assessment Test (FCAT)-exempt and MJ-3 pre- 
algebra students exempt from the FCAT, are reported in Appendix A4, but are not included in the WWC rating of effectiveness for the intervention. 

9. The four courses in the Kerstyn (2001; 2002) studies— Algebra 1, Algebra 1 Flonors, MJ-3 pre-algebra, and MJ-3 Advanced— were treated as separate 
studies because they examined effects of / CAN Learn'^ on different samples of students using different curricula. 
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effects, potentially negative, or negative. The rating of effective- the difference between participants in the intervention and the 

ness takes into account four factors: the quality of the research comparison conditions, and the consistency in findings across 

design, the statistical significance of the findings, the size of studies (see the WWC Intervention Rating Scheme). 



The WWC found / CAN Learn® 
Pre-Algebra and Algebra 
to have positive effects for 
math achievement 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations). The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is based entirely on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable 
to the intervention group. 



The average improvement index for math achievement is +5 
percentile points across the 11 samples in the five studies, with 
a range of -7 to +16 percentile points across findings. 

Summary 

The WWC reviewed 27 studies on / CAN Learn® Pre-Algebra 
and Algebra. One of these studies meets WWC evidence 
standards; four studies meet WWC evidence standards with 
reservations; the remaining 22 studies do not meet either WWC 
evidence standards or eligibility screens. Based on the five 
studies, the WWC found positive effects in math achievement. 
The conclusions presented in this report may change as new 
research emerges. 
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Appendix A1.1 


Study Characteristics: Kirby, 2006 (randomized controiied triai) 


Characteristic 


Description 


Study citation 


Kirby, P. C. (2006). / CAN Leanf in Orleans Parish Public Schools: Effects on LEAP 8th grade math achievement, 2003-2004. (New Orleans, LA: Ed-Cet, Inc.) 
Additional source: Kirby, P. C. (2004c). / CAN LearrP in Orleans Parish Public Schools: Effects on LEAP 8th grade math achievement, 2003-2004. (New Orleans, LA: 
Ed-Cet, Inc.) 


Participants 


The study sample included 2,400 eighth-grade students (1,082 / C4A/ /.earn® students and 1,318 traditional mathematics students) and 57 math teachers (29 teachers 
taught only traditional classes; 15 taught only / CANLearrP classes; 13 taught both classes) in 13 Orleans Parish Public Schools.'' After school clerks entered student names 
and courses, the Pearson SASI® Basic Scheduling (3rd edition) software program used a random assignment algorithm to schedule the students into intervention and com- 
parison classes. Teachers were not randomized to conditions. Only students with no special education classification were included in the analyses. About 96% of fhe students 
in the / CAN Learrf' and traditional group were African-American, less than 1% were Hispanic, and less than 0.5% were Caucasian.^ Of these students, approximately 50% 
of the / CAN LearrF group and 50% of the traditional class group were female. 


Setting 


The participating students were from the Orleans Parish public school system, which includes the city of New Orleans. The participating schools were selected for this study 
based on two criteria: the schools included both traditional and / CANLearrP eighth-grade classes, and those classes included at least 20 students each. 


Intervention 


Students were taught using the / CAN LearrP mathematics curriculum. According to the author, / CAN LearrP students are expected to complete about 100 lessons during 
a school year. However, intervention students in this study began using / CAN LearrP in the second semester of fhe 2003-04 school year. Students completed 12.1 lessons 
on average, with time spent on / CAN Learrf’ ranging from using only the test prep module to completing 95 lessons. The intervention and evaluation occurred during the 
2003-04 academic year. 


Comparison 


Comparison students were taught in traditional classes with the teacher serving as the primary deliverer of instruction. The author did not provide further information on the 
curriculum. 


Primary outcomes 
and measurement 


The primary outcome measure was the Louisiana Educational Assessment Program (LEAP) Grade 8 Mathematics Exam administered in spring 2004. Eor a more detailed 
description of this outcome measure, see Appendix A2. 


Staff/teacher training 


Information on teacher training was not provided. 



1 . The original sample included 14 schools. However, random assignment occurred in only 13 schools. For the purposes of this review, the study design and findings are reported for the subsample 
of 13 schools. Further, the study author conducted analyses of subsamples of students in addition to the total sample of 2,400 students in 13 schools. Those analyses focused on students of 
teachers who taught both conditions and students with matched pretest and posttest scores. 

2. The study notes that these data were collected in the Orleans Parish Public Schools before Hurricane Katrina. The demographics in the school district have changed since the study was conducted. 
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Appendix A1.2 Study Characteristics: Kerstyn, 2001 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Kerstyn, C. (2001). Evaluation of the 1 CAN LearrP mathematics classroom: First year of implementation (2000-2001 school year). (Available from the Division of Instruction, 
Hillsborough County Public Schools, 901 East Kennedy Blvd., Tampa, FL 33602.) 


Participants 


The study sample^ included 2,536 eighth-grade students in 116^ classes (58 / CANLearrF classes with 1,222 students, and 58 traditional classes with 1,314 students) in 
36 Title 1 middle schools in the Hillsborough County Public School District in Florida. The study was limited to regular education students. All 1 1 6 classes were used in the 
analysis. Approximately 49% of the total sample were males (50% / CAN Learrf, A7% comparison) and 51 % were females (50% / CAN LearrP, 53% comparison). Roughly 
47% of the total sample qualified for the National School Lunch Program (49% / CAN LearrP, 44% comparison); 42% of the total sample were Caucasian (37% / CAN Learrf, 
47% comparison); 27% were African-American (30% / CAN Learrf’, 25% comparison); 31% were Hispanic (33% / CANLearrP, 28% comparison) and 5% were of other racial 
classification (4% / CAN Learrf, 5% comparison). Within subgroups, a larger number of students in the / CAN Learrf’ classes were at the pre-algebra and algebra levels. 


Setting 


The participating students were from middle schools in the Hillsborough County Public School system in Florida, which includes the Tampa metro area. 


Intervention 


Students were taught using the / CAN LearrP mathematics curriculum. The author did not indicate how many of the lessons were required to be completed for the curriculum 
to be implemented as intended. The author indicated that the / CAN LearrP system was implemented in 45-, 50-, 80-, and 90-minute class periods. The intervention and 
evaluation occurred during the 2000-01 academic year. 


Comparison 


Comparison students were taught using a traditional instructional method. The author did not provide further information on the curriculum. 


Primary outcomes 
and measurement 


The primary outcome measure was the Florida Comprehensive Assessment Test (FCAT) Grade 8 Math Test administered in February 2001 For a more detailed description 
of this outcome measure, see Appendix A2. 


Staff/teacher training 


Teachers in this study participated in training sessions on the use of the software and hardware, but not on use of the software in instruction. 



1 . The reported study sample was taken from Table 5 (page 9 of the study), while the reported demographic Information was taken from Table 4 (page 8 of the study). 

2. Originally the study included 59 matched pairs of / CAN Learn* and traditional classes, for a total of 118 classes; however, one of the pairs (MJ-3 #12) was dropped due to the traditional class- 
room transitioning to be an / CAN Learn* classroom in the second semester. 

3. A Semester 1 exam was also administered to study participants at the end of the first semester. The results are not reported in Appendix A3 with the results of the FCAT, because the psycho- 
metric properties of the Semester 1 exam were not reported, and it measured the effect of the intervention at an earlier point than did the FCAT. Because the FCAT measured math achievement 
at the end of the school year, the WWC reasoned that the FCAT was a more appropriate and valid measure of the intervention’s end-of-year effects. 
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Appendix A1.3 Study Characteristics: Kerstyn, 2002 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Kerstyn, C. (2002). Evaluation of the 1 CAN LearrP mathematics classroom: Second year of implementation (2001-2002 school year). (Available from the Division ot Instruction, 
Hillsborough County Public Schools, 901 East Kennedy Blvd., Tampa, FL 33602.) 


Participants 


The study sample included 11,125 eighth-grade students in 597 classes (129 / CANLearrP classes with 1,871 students and 468 traditionally taught mathematics classes 
with 9,254 students) enrolled in Algebra 1, Algebra 1 Honors, MJ-3 pre-algebra, or MJ-3 Advanced math classes in the 36 middle schools in the Hillsborough County School 
District in Florida. Approximately 50% of the total sample were males (47% / CANLearrf, 51% comparison) and 50% were females (53% / CAN Learrf, 49% comparison). 
Roughly 44% of the total sample qualified for the National School Lunch Program (65% / CAN Learrf, 39% comparison); 50% of the total sample were Caucasian (34% / 
CANLearrP, 53% comparison); 21% were African-American (28% / CAN LearrP, 20% comparison); 24% were Hispanic (34% / CAN Leanf, 22% comparison) and 5% were 
of other racial classifications (4% / CANLearrf, 5% comparison). Compared with classes using a traditional instruction method, a higher proportion of / CAN Learif students 
were in the free and reduced-price lunch program and were from minority backgrounds (African-American and Hispanic). 


Setting 


The participating students were from middle schools in the Hillsborough County Public School system in Florida, which includes the Tampa metro area. 


Intervention 


Students were taught using the / CANLearrP Algebra curriculum, which consists of 109 complete algebra lessons that students complete individually at their desks by 
accessing software through a number of DVD disks. The / CAN Learrf system is intended to be the primary source of instruction. The intervention and evaluation occurred 
during the 2001-02 academic year. 


Comparison 


Comparison students were taught using a traditional instruction method. The author did not provide further information on the method. 


Primary outcomes 
and measurement 


The primary outcome measure was the Florida Comprehensive Assessment Test (FCAT) Grade 8 Math Test administered in 2002. For a more detailed description of this 
outcome measure, see Appendix A2. 


Staff/teacher training 


Information on teacher training was not provided. 
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Appendix A1.4 Study Characteristics: Kirby, 2004a (randomized controiied triai with teacher-intervention confound probiem) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2004a). Comparison of! CAN Leartf and traditionally-taught 8th grade general math student performance on the California Standards Test, Spring 2004. (New 
Orleans, LA: Ed-Cet, Inc.) 


Participants 


The study sample included 204 eighth-grade students enrolled in General Mathematics (91 in / CAN Learrf classes and 113 in traditionally-taught classes) in Bret Harte 
Middle School. The / CAN LearrP classes contained a higher proportion of African-American sfudents, and a lower proportion of Hispanic and non-nafive English speaking 
studenfs, than the comparison classes. Approximately 59% of the total sample were males (56% / CAN Learrf, 62% comparison) and 41% were females (44%o / CAN LearrP, 
38% comparison); 36% of the total sample were African-American (48%o / CANLearrf, 26% comparison), 29% were Hispanic (22% / CAN Learrf, 35% comparison), and 
35% were reported as “all other” (30% / CANLearrP, 39% comparison). 


Setting 


The participating students were from Bret Harte Middle School, which is one of five middle schools in Hayward Unified School District in Alameda County, California. 


Intervention 


Students were taught eighth-grade mathematics by one teacher using the / CAN LearrP Pre-algebra curriculum.^ The / CAN LearrP lessons were chosen to align to the Cali- 
fornia Mathematics Standards. The teachers used the / CAN LearrP computer system as the primary system of instructional delivery and used the Classroom Manager class 
management/grade book system to identify students who needed additional instruction on specific concepts. The intervention and evaluation occurred during the 2003-04 
academic year. 


Comparison 


Comparison students were taught in traditional classes, with the teacher as the primary deliverer of instruction, using a curriculum based on the state-adopted Glencoe 
pre-algebra textbook. 


Primary outcomes 
and measurement 


The primary outcome measure was the General Mathematics exam from the California Standards Test (CST) administered in 2004. For a more detailed description of this 
outcome measure, see Appendix A2. 


Staff/teacher training 


The intervention teacher received professional development in using the / CAN Learrf’ software and managing the instructional environment. 



1 . The study authors provided the WWC with the number of teachers in each condition 
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Appendix A1.5 Study Characteristics: Kirby, 2004b (randomized controiied triai with teacher-intervention confound probiem) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2004b). Comparison of 1 CAN LearrP and traditionally-taught 8th grade student performance on the Georgia Criterion-Referenced Competency Test New Orleans, 
LA: Ed-Cet, Inc. 


Participants 


The study sample included 254 eighth-grade students (91 in / CAN LearrP classes and 163 in traditional classes) in Gilmer Middle School in Ellijay, Georgia who were not 
classified to receive special education or gifted services. School staff first stratified sample students based on prior year aohievement and then randomly assigned the 
students to / CAN Learrf or traditional classes. Approximately 49% of the total sample were males (45% / CAN Learrf, 52% comparison) and 51% were females (55% / CAN 
LearrP, 48% comparison); 47% of the sample qualified for the National Sohool Lunch Program (41% / CAN LearrP, 50% comparison); 93% were Caucasian (95% / CAN 
LearrP, 91% comparison), 0.5% were African-American (0% / CAN LearrP, 1% comparison), and 7% were Hispanic (6% / CANLearrf, 8% comparison). 


Setting 


The partioipating students were from one middle sohool in the Gilmer County School District in northwestern Georgia. 


Intervention 


Students were taught by one teaoher using the / CAN LearrP mathematics curriculum. The intervention teaoher used the / CANLearrf computer system as the primary system 
of instruotional delivery and used the Classroom Manager olass management/grade book system to identify students who needed additional instruction on specifio concepts. 
The intervention and evaluation oocurred during the 2003-04 aoademic year. 


Comparison 


Comparison students were taught in traditional olasses, with the teacher serving as the primary deliverer of instruction. The author did not provide further information on the ourrioulum. 


Primary outcomes 
and measurement 


The primary outcome measure was the Georgia Criterion-Referenced Competency Test (GCRCT) Math Test administered in 2004. Eor a more detailed description of this 
outcome measure, see Appendix A2. 


Staff/teacher training 


The intervention teacher received professional development in using the / CAN Learrf software and managing the instructional environment. 
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Appendix A2 Outcome measures for the math achievement domain 



Outcome measure 


Description 


Florida Comprehensive 
Assessment Test (FCAT) 
Grade 8 Math Test 


The FCAT math test is a standardized measure that inciudes items reiated to all five content strands of Fiorida's Sunshine State Standards tor mafhemafics: number sense, 
concepts, and operations; measurement; geometry and spatial sense; algebraic thinking; and data analysis and probability (as cited in Kerstyn, 2001; Kerstyn, 2002). Test 
content at grade 8 is evenly divided among these five content strands. Students are given 160 minutes to take the exam, which includes multiple-choice items, gridded- 
response items, and performance tasks. Test results are reported as scale scores, which range from 100 to 500. 


General Mathematics 
exam from the California 
Standards Test (CST) 


The General Mathematics CST for grade 8 is based on the California Mathematics Standards for grades 6 and 7 (as cited in Kirby, 2004a). The CST is administered to 
students in grades 8 and 9 who have not yet completed or are not enrolled in discipline-specific, standards-based math courses or who are enrolled in the first year of a multi- 
year Algebra 1 course. The Mathematics CST consists of 65 multiple-choice questions. Test results are reported as scale scores, which range from 150 to 600. 


Louisiana Educational 
Assessment Program (LEAP) 
Grade 8 Mathematics Exam 


The LEAP Mathematics test for grade 8 includes a multiple-choice section and an open-ended section for more complex tasks (as cited in Kirby, 2006). The test is aligned 
to the state's Comprehensive Curriculum and Grade Level Expectations. Six strands of the Louisiana Mathematics Framework are represented in LEAP: number and number 
relations; algebra; measurement; geometry; data analysis, probability, and discrete math; and patterns, relations, and functions. Test results are reported as scale scores, 
which range from 100 to 500. 


Georgia Criterion- 
Referenced Competency 
Test (GCRCT) Math Test 


The GCRCT is designed to measure how well students acquire the skills and knowledge described in the Georgia Performance Standards and the Quality Core Curriculum (as 
cited in Kirby, 2004b). The GCRCT Math Test contains 60 items in six areas: number sense and numeration; geometry and measurement; patterns and relationships/algebra; 
statistics and probability; computation and estimation; and problem solving. Test results are reported as scale scores, which range from 150 to 450. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 



Authors’ findings from the study 



Outcome measure 


Study 

sample 


Sample size 
(clusters/ 
students) 


Mean outcome 
(standard deviation)^ 




WWC calculations 




ICANLeanf’ 

group 


Comparison 

group 


Mean 

difference^ 
{ICANLearrf^ 
- comparison) 


Effect 

size^ 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 










Kirby, 2006 (randomized controlled trial)^ 








LEAP Math 


Grade 8 


13/2,400 


295.30 


278.82 


16.48 


0.35 


Statistically 


+14 


scale scores 






(42.26) 


(49.82) 






significant 




Average for math achievement (Kirby, 2006)^ 










0.35 


Statistically 


+14 
















significant 








Kerstyn, 2001, Algebra 1 (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


16/350 


351.38^ 


344.469 


6.92 


0.23 


ns 


+9 








(30.80) 


(28.36) 










Average for math achievement (Kerstyn, 2001, Algebra 1)^ 








0.23 


ns 


+9 






Kerstyn, 2001, Algebra 1 Honors (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


16/336 


372.99*’ 


373.739 


-0.74 


-0.02 


ns 


-1 








(34.47) 


(35.80) 










Average for math achievement (Kerstyn, 2001, Algebra 1 Honors)^ 








-0.02 


ns 


-1 






Kerstyn, 2001, MJ-3 pre-algebra (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


64/1,420 


296.77*’ 


293.899 


2.88 


0.08 


ns 


+3 








(38.23) 


(38.09) 










Average for math achievement (Kerstyn, 2001, MJ-3 pre-algebra)^ 








0.08 


ns 


+3 






Kerstyn, 2001, MJ-3 Advanced (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


20/430 


332.519 


327.409 


5.11 


0.17 


ns 


+7 








(31.19) 


(29.60) 










Average for math achievement (Kerstyn, 2001, MJ-3 Advanced)^ 








0.17 


ns 


+7 



(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(clusters/ 
students) 


/ CAN Learif Comparison 

group group 


Mean 

difference^ 
{ICANLearif> 
- comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



FCAT mathematics 


Grade 8 


Kerstyn, 2002, Algebra 1 (quasi-experimental design)^ 

66/1,195 347 , 431 “ 349,7911 -2,36 

(20,88) (21,04) 


-0,11 


ns 


-5 


Average for math achievement (Kerstyn, 2002, Algebra 1)^ 








-0.11 


ns 


-5 






Kerstyn, 2002, Algebra 1 Honors (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


91/1,894 


369.58“ 


374.311“ 


-4.73 


-0.17 


ns 


-7 








(27,84) 


(27.61) 










Average for math achievement (Kerstyn, 2002, Algebra 1 Honors)^ 








-0.17 


ns 


-7 






Kerstyn, 2002, MJ-3 pre-algebra (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


328/5,957 


293,81“ 


289.901“ 


3.91 


0.10 


Statistically 


+4 








(31.96) 


(40.55) 






significant 




Average for math achievement (Kerstyn, 2002, MJ-3 pre-algebra)^ 








0.10 


Statistically 


+4 
















significant 








Kerstyn, 2002, MJ-3 Advanced (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


112/2,079 


329.93“ 


331 . 041 “ 


-1.11 


-0.05 


ns 


-2 








(23.30) 


(24.08) 










Average for math achievement (Kerstyn, 2002, MJ-3 Advanced)^ 








-0.05 


ns 


-2 






Kirby, 2004a (randomized controlled trial with teacher-intervention confound problem)^ 






Generai Mathematics CST 


Grade 8 


1/204 


315.58 


299.73 


15.85 


0.33 


Statistically 


+13 








(45.75) 


(49.71) 






significant 




Average for math achievement (Kirby, 2004a)^ 










0.33 


Statistically 


+13 
















significant 





(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(clusters/ 
students) 


/ CAN Learti^ Comparison 

group group 


Mean 

difference^ 
{ICANLearnf^ 
- comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Kirby, 2004b (randomized controlled trial with teacher-intervention confound problem)^ 



GCRCr^ Grade 8 1/254 


333.54 

(35.72) 


319.89 

(31.73) 


13.65 


0.41 


Statistically 

significant 


+16 


Average for math achievement (Kirby, 2004b)^ 








0.41 


Statistically 

significant 


+16 


Oomain average for math achievement across all studies^ 








0.12 


na 


+5 



ns = not statistically significant 
na = not applicable 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. Subgroup findings from the same studies are not 
included in these ratings, but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see Technical Details of 
WWC-Conducted Computations. In the cases of Kerstyn (2001) and Kirby (2006), corrections for clustering were needed, so the significance levels may differ from those reported in the original 
studies. No other studies required corrections for clustering or multiple comparisons. 

8. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 

9. For Kerstyn (2001), the author reported both classroom-level and student-level posttest means and standard deviations. In order to compute a student-level effect size that could be considered 
for rating purposes, the WWC used the student-level data and corrected for clustering. For further details, please see Technical Details of WWC-Conducted Computations. 

10. The intervention group value from Kerstyn (2002) is the intercept from the hierarchical linear modeling (HLM) model plus the program coefficient from the HLM analysis. 

11. The control group mean from Kerstyn (2002) is the intercept from the HLM model. 

12. The author reported results from the Georgia Criterion-Referenced Competency Test as scale scores and as criterion scores (that is, the percentage that passed the criterion score compared 
with the percentage that failed), but the WWC focused on the results from the scale scores because they contain more information than categorical scores. 
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Appendix A4 Summary of subgroup findings for the math achievement domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(clusters/ 
students) 


/ CAN Learif‘ Comparison 

group group 


Mean 

difference^ 
(ICANLearif" 
- comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



FCAT mathematics 


Grade 8 


Kerstyn, 2002, MJ-3 pre-algebra standard curriculum^ (quasi-experimental design)^ 

258/4,045 294.169 295.06^9 -0,90 -0,03 

(33.46) (33.91) 


ns 


-1 


FCAT mathematics 


Grade 8 


Kerstyn, 2002, MJ-3 pre-algebra FCAT-exempt^ (quasi-experimental design)^ 

249/888 299.539 284.3819 15.15 0.29 


Statistically 


+12 






(37.62) (52.87) 


significant 





ns = not statistically significant 



1. This appendix presents subgroup findings for measures that fall in math achievement. The standard curriculum and FCAT-exempt subgroup scores from Kerstyn (2002) are presented in Appendix 
A4, and these combined scores can be found in Appendix A3. Total group scores were used for rating purposes and are presented in Appendix A3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. Standard curriculum and FCAT-exempt groups are categorized by the state in terms of their impact on school accountability grades. The standard curriculum students are used in the school 
accountability grade, while the FCAT-exempt students (including ESE, LEP, and Homebound students) are not. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas 
the WWC used to calculate statistical significance, see Technical Details of WWC-Conducted Computations. In the case of Kerstyn (2002), no correction for clustering was needed. 

9. The intervention group value from Kerstyn (2002) is the intercept from the HIM model plus the program coefficient from the HLM analysis. 

10. The control group mean from Kerstyn (2002) is the intercept from the HLM model. 
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Appendix A5 / CAN Leanfi Pre-Algebra and Algebra rating for the math achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math achievement, the WWC rated / CAN Learn® Pre-Algebra and Algebra as positive. The remaining ratings (potentiaiiy positive, mixed, 
no discernible effects, potentially negative, and negative) were not considered, as / CAN Learn® was assigned the highest applicable rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Four studies of / CAN Learn® showed statistically significant positive effects. Of those, one study had a strong design according to WWC 
standards. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. Four studies of / CAN Learn® showed statistically significant positive effects. The remaining study showed indeterminate effects. No studies 
showed statistically significant or substantively important negative effects. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. 
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Appendix A6 Extent of evidence by domain 









Sample size^ 




Outcome domain 


Number of studies 


Schools 


Students 


Extent of evidence^ 


Math achievement 


5 


51 


16,519 


Medium to large 



1 . The samples in Kerstyn (2001) and Kerstyn (2002) comprised distinct samples of students within the same set of 36 schools. 

2. A rating of “medium to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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